Depth sensing systems and methods

ABSTRACT

A depth sensing system includes a sensor having first and second sensor pixels to receive light from a surface. The system also includes a filter to allow transmission of full spectrum light to the first sensor pixel and visible light to the second sensor pixel while preventing transmission of infrared light to the second sensor pixel. The system further includes a processor to analyze the full spectrum light and the visible light to determine a depth of the surface. The filter is disposed between the sensor and the surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. application Ser. No.16/737,765, entitled “DEPTH SENSING SYSTEMS AND METHODS,” filed Jan. 8,2020, under attorney docket number ML-0304USDIV1, which is a divisionalof U.S. application Ser. No. 15/445,818, entitled “DEPTH SENSING SYSTEMSAND METHODS,” filed Feb. 28, 2017, which claims priority to U.S.Provisional Application Ser. No. 62/301,847, filed on Mar. 1, 2016 andentitled “DEPTH SENSING SYSTEMS AND METHODS.” The present application isrelated to U.S. Provisional Patent Application Ser. No. 62/012,273,filed on Jun. 14, 2014 and U.S. Utility patent application Ser. No.14/738,877 filed on Jun. 13, 2013. The contents of the aforementionedpatent applications are hereby expressly and fully incorporated byreference in their entirety, as though set forth in full. Described inthe patent application attorney Ser. Nos. 62/012,273 and 14/738,877 arevarious embodiments of virtual reality and augmented reality systemswherein projected light is used in depth sensing. Described herein arefurther embodiments of projected light depth sensing systems and lightsensors for depth sensing.

FIELD OF THE INVENTION

The present disclosure relates to depth sensing systems and methods.

BACKGROUND

Depth sensing is the determination of the distance between a known pointin three dimensional (“3D”) space (e.g., a sensor) and a point ofinterest (“POI”) on a surface of an object. Depth sensing is also knownas texture sensing because determining the respective distances of aplurality of POIs on a surface determines the texture of that surface.Depth or texture sensing is useful for many computer vision systems,including mixed reality systems.

Modern computing and display technologies have facilitated thedevelopment of mixed reality systems for so called “virtual reality” or“augmented reality” experiences, wherein digitally reproduced images orportions thereof are presented to a user in a manner wherein they seemto be, or may be perceived as, real. A virtual reality, or “VR”,scenario typically involves presentation of digital or virtual imageinformation without transparency to actual real-world visual input. Anaugmented reality, or “AR”, scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user (i.e., transparency to other actualreal-world visual input). Accordingly, AR scenarios involve presentationof digital or virtual image information with transparency to otheractual real-world visual input.

Various optical systems generate images at various depths for displayingmixed reality (VR and AR) scenarios. Some such optical systems aredescribed in U.S. Utility patent application Ser. No. 14/738,877, thecontents of which have been previously incorporated-by-reference herein.Other such optical systems for displaying mixed reality scenarios aredescribed in U.S. Utility patent application Ser. No. 14/555,585 filedon Nov. 27, 2014, the contents of which are hereby expressly and fullyincorporated by reference in their entirety, as though set forth infull.

AR scenarios often include presentation of virtual image elements inrelationship to real-world objects. For example, referring to FIG. 1, anaugmented reality scene 100 is depicted wherein a user of an ARtechnology sees a real-world park-like setting 102 featuring people,trees, buildings in the background, and a concrete platform 104. Inaddition to these items, the user of the AR technology also perceivesthat he “sees” a robot statue 106 standing upon the real-world platform104, and a cartoon-like avatar character 108 flying by which seems to bea personification of a bumble bee, even though these elements 106, 108do not exist in the real world. In order to present a believable orpassable AR scene 100, the depth of real world objects (e.g., theplatform 104) must be determined to present virtual objects (e.g., therobot statue 106) in relation to real world objects.

VR scenarios that include reproduction of portions of real worldenvironments can also benefit from determination of the depth andtexture of those portions of the real world environment. Accurate depthand texture information will result in more accurate VR scenarios. BothAR and VR scenarios may also include outwardly directed cameras tocapture portions of real world environments (e.g., for analysis ortransmission). Focusing these outwardly directed cameras can be aided bydetermination of the depth of those portions of the real worldenvironment.

One approach to depth sensing includes measuring the respective anglesbetween the optical axes of two images (which are separated by a knowndistance at a known orientation) of a single POI on a surface and thePOI on the respective images. Then determining the depth of the surfaceby triangulating the measured angles and the known distance between theimage capture locations. Problems with this approach include (1)identification of the POI (especially on a homogenous surface)(“identification problem”) in the first image and (2) identification ofthe corresponding POI in the second image (‘correspondence problem’).The systems and methods described herein are configured to address thesechallenges.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users.

In one embodiment, a depth sensing system includes a sensor having firstand second sensor pixels to receive light from a surface. The systemalso includes a filter to allow transmission of full spectrum light tothe first sensor pixel and visible light to the second sensor pixelwhile preventing transmission of infrared light to the second sensorpixel. The system further includes a processor to analyze the fullspectrum light and the visible light to determine a depth of thesurface. The filter is disposed between the sensor and the surface.

In one or more embodiments, the sensor has a plurality of second sensorpixels including the second sensor pixel, each of the plurality ofsecond sensor pixels is adjacent the first sensor pixel, and the filterallows transmission of visible light to each of the plurality of secondsensor pixels while preventing transmission of infrared light to each ofthe plurality of second sensor pixels. Analyzing the full spectrum lightand the visible light may include calculating an estimated visible lightvalue for the first sensor pixel based on a plurality of detectedvisible light values corresponding to the plurality of second sensorpixels. Calculating the estimated visible light value may includeaveraging the plurality of detected visible light values. Calculatingthe estimated visible light value may include performing edge detectionon the plurality of detected visible light values.

In one or more embodiments, the sensor has a plurality of first sensorpixels including the first sensor pixel, each of the plurality of firstsensor pixels is adjacent the first sensor pixel, and the filter allowstransmission of visible light to each of the plurality of second sensorpixels while preventing transmission of infrared light to each of theplurality of second sensor pixels. Analyzing the full spectrum light andthe visible light may include calculating a plurality of estimatedvisible light values for each of the plurality of first sensor pixels,and calculating an estimated visible light value for the first sensorpixel based on at least some of the plurality of estimated visible lightvalues.

In one or more embodiments, the system also includes a spatiallymodulated light projection device to project the light toward thesurface, where the light is reflected from the surface toward thesensor. Analyzing the full spectrum light and the visible light mayinclude generating an infrared light image of the surface. Analyzing thefull spectrum light and the visible light may include triangulating aPOI in the infrared light image of the surface.

In another embodiment, a depth sensing system includes a spatiallymodulated light projection device to project light toward a surface. Thesystem also includes a sensor to receive the light reflected from thesurface. The system further includes an actuator to control spatialmodulation of the light projection device and to receive lightinformation from the sensor. Moreover, the system includes a processorto analyze the light reflected from the surface to determine a depth ofthe surface. The light includes infrared light having a wavelength fromabout 700 nm to about 1 mm.

In one or more embodiments, the light includes visible light having awavelength from about 390 nm to about 700 nm. The actuator may controlspatial modulation of the light projection device to form a patternincluding the infrared light and the visible light. The actuator maycontrol spatial modulation of the light projection device to form twointersecting line segments on the surface. The actuator may controlspatial modulation of the light projection device to form a staticpattern including two intersecting line segments on the surface. Theactuator may control spatial modulation of the light projection deviceto form a dynamic pattern including two intersecting line segments onthe surface. The actuator may control spatial modulation of the lightprojection device to form a pattern including a plurality of discreteintersecting line segments on the surface.

In one or more embodiments, controlling spatial modulation of the lightprojection device includes controlling movement of at least a portion ofthe light projection device. Controlling spatial modulation of the lightprojection device may include controlling projection of the light by thelight projection device. The spatially modulated light projection devicemay include a fiber scanned display. The spatially modulated lightprojection device may include a laser light source. The spatiallymodulated light projection device may include a laser scanning display.

In still another embodiment, a depth sensing system includes a sensorhaving first and second sensor pixels to receive light from a surface.The system also includes a filter to allow transmission of a firstproportion of visible light and a second proportion of infrared light tothe first sensor pixel and a third proportion of visible light and afourth proportion of infrared light to the second sensor pixel. Thefirst proportion of visible and the second proportion of infrared lightresult in a first sensed value at the first sensor pixel. The thirdproportion of visible and the fourth proportion of infrared light resultin a second sensed value at the second sensor pixel. The system furtherincludes a processor to analyze the first and second sensed values todetermine a depth of the surface. The filter is disposed between thesensor and the surface.

In one or more embodiments, analyzing the first and second sensed valuesincludes generating an infrared light image of the surface. Analyzingthe first and second sensed values may include triangulating a POI inthe infrared light image of the surface.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of various embodiments ofthe present invention. It should be noted that the figures are not drawnto scale and that elements of similar structures or functions arerepresented by like reference numerals throughout the figures. In orderto better appreciate how to obtain the above-recited and otheradvantages and objects of various embodiments of the invention, a moredetailed description of the present inventions briefly described abovewill be rendered by reference to specific embodiments thereof, which areillustrated in the accompanying drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 depicts a user's view of augmented reality (AR) through awearable AR user device according to one embodiment;

FIGS. 2 to 4 are detailed views of line segments and patterns projectedby various prior art depth sensing systems;

FIGS. 3 and 4 also depict scan areas of various prior art depth sensingsystems;

FIGS. 5 to 9 are detailed views of line segments and patterns projectedby depth sensing systems according to various embodiments;

FIG. 10 is a detailed schematic view of a depth sensing system accordingto one embodiment;

FIG. 11 is a detailed perspective view of the depth sensing systemdepicted in FIG. 10 in use;

FIG. 12 is a detailed schematic view of a depth sensing system accordingto another embodiment;

FIG. 13 is a detailed schematic view of a prior art depth sensingsystem;

FIGS. 14 and 15 are detailed schematic views of depth sensing systemsaccording to two embodiments;

FIGS. 16-18 are detailed schematic views of a light sensor for depthsensing systems according to various embodiments;

FIGS. 19 and 21 are flowcharts depicting image processing methodsaccording to two embodiments;

FIG. 20 is a detailed schematic view of a light sensor for depth sensingsystems according to another embodiment.

DETAILED DESCRIPTION

Various embodiments of the invention are directed to systems, methods,and articles of manufacture for depth sensing systems in a singleembodiment or in multiple embodiments. Other objects, features, andadvantages of the invention are described in the detailed description,figures, and claims.

Various embodiments will now be described in detail with reference tothe drawings, which are provided as illustrative examples of theinvention so as to enable those skilled in the art to practice theinvention. Notably, the figures and the examples below are not meant tolimit the scope of the present invention. Where certain elements of thepresent invention may be partially or fully implemented using knowncomponents (or methods or processes), only those portions of such knowncomponents (or methods or processes) that are necessary for anunderstanding of the present invention will be described, and thedetailed descriptions of other portions of such known components (ormethods or processes) will be omitted so as not to obscure theinvention. Further, various embodiments encompass present and futureknown equivalents to the components referred to herein by way ofillustration.

The depth sensing systems may be implemented independently of mixedreality systems, but many embodiments below are described in relation toAR systems for illustrative purposes only.

Summary of Problem and Solution

There are various methods for optically estimating or sensing the depthof a surface (i.e., the distance between a known point in 3D space and apoint on the surface). Depth sensing methods can be classified into twoprimary modes: passive and active. Passive systems detect ambient lightfrom light sources outside of the systems (e.g., overhead lights or thesun) that is reflected by the surface. Active systems project light ontoa surface and detect the projected light reflected by the surface.

Passive systems typically determine depth using two images captured atdifferent locations that are separated by a known distance. Some passivesystems capture the two images using multiple cameras (as in a binocularconfiguration). Other passive systems capture the two images using thesame sensor at different times and locations. After the two images havebeen captured at the different locations, the systems process the imagesto match a POI in one image to the corresponding POI in the other image.Then, the systems triangulate the angles between the optical axes of thetwo images and a single POI on a surface, and the distance between thetwo image capture locations to determine the location of the POI in 3Dspace relative to the two image capture locations, which are known tothe system.

Passive systems can determine the location of a POI in 3D space, buthave multiple failure modes, including the lack of suitable POIs thatcan be identified in one image (“identification problem”) and matchedwith their counterparts in the other image (“correspondence problem”).An example of this failure mode is imaging a blank white wall, in which(1) identifying a point on the wall in one image and (2) identifying thecorresponding point in the other image are exceedingly difficult.Similarly, in the case of a dark room, there simply isn't enough ambientlight to clearly see objects, and thus, identifying and matching them isalso exceedingly difficult.

Some active systems address these two problems (i.e., lack of light andlack of distinguishable features) by projecting patterned or texturedlight (e.g., light from a kaleidoscope). Such systems illuminate thesurface and project a pattern over a homogenous surface (e.g., a whitewall). When two images are captured using such active systems with astatic illumination pattern (by either simultaneous capture or twocameras with stationary projection), it is much simpler to matchpatterns (or portions thereof) on the surface from one image to theother. Therefore, triangulating the location of POIs in 3D space iscorrespondingly simpler. In fact, using an advanced system with tightmechanical tolerances, a single image captured by a single camera can beused to determine the location of POIs, because the location of the POIin 3D space can be calculated by triangulating both the angle of theobserved reflection and the locations of the camera and the lightprojector.

However, even with active systems, the use of visible light is oftensuboptimal, because it can be distracting or disorienting to users orothers near the surface. Some active systems address this problem byprojecting infrared (“IR”) light, which modern raw camera sensors candetect (e.g., near-infrared photons), but the human eye cannot see.

Even using infrared active systems, other patterns in the scene mayinterfere with the depth sensing or causes other depth sensing problems.Other systems include a wavelength filter disposed over the sensor(which natively can sense both visible and infrared light), such thatvisible light does not reach the sensor. Adding the filter results indetection of an infrared-only image, which is typically only illuminatedby the system (infrared) projector, the sun and a few other sources ofinfrared light (e.g., warm light bulbs and remote controls).

To obtain the most accurate (i.e., closest to the true value) andprecise (i.e., reproducible) depth information, a very high resolutionsensor is desired. Greater numbers of sensor pixels over the viewingarea results in a reduction in the angular resolution of each sensorpixel, effectively providing a higher precision angular input into thetriangulation mathematics. As used in this application, “sensor pixel”includes, but is not limited to, distinguishable points on a sensor formeasurement of light intensity.

Infrared active systems require sensors/sensor pixels that detectinfrared light in addition to sensors/sensor pixels that detect visiblelight in systems including outwardly directed cameras. Problems withthis approach include (1) reduction of visible light resolution insystems including hybrid red/green/blue/infrared sensors and (2)coordinate registration in system including separate visible andinfrared sensors. The embodiments disclosed herein address these andother sensor problems by using an improved image sensor with augmenteddepth sensing, as described below.

Without additional processing, the maximal angular precision for a POIis achieved with an angular resolution of a single sensor pixel.However, image processing algorithms can provide “subpixel resolution.”For instance, the system can observe some features (e.g., lines makingup the corner edge of a desk) over the course of several sensor pixels,and reconstruct the equation of the line at a precision higher than thatof a single sensor pixel.

Depth sensing methods are subject to the Nyquist limits of the systemsbecause they are signal processing methods. As such, a minimum amount ofsampling points (and point density) based on the signal frequency isrequired to reconstruct a signal. Therefore, noisier signals are harderto sample and reconstruct than “simpler” (lower bandwidth) signals. Theembodiments disclosed herein address the Nyquist limit associatedproblems and other projector related problems by using a non-aliasingpattern projector/generator to reduce the high frequency noise of theprojected light, as described below.

Dynamic Non-Aliasing Pattern Projector/Generator

Fiber Scan Projectors (“FSP”) project light by selectively vibrating afiber-optic tip at high frequencies. FSP are also known as fiber scanneddisplays (“FSD”), fiber scanning displays (“FSD”), scanned fiberdisplays and scanning fiber displays. FSPs can project a dynamic imageas a display for human observation. Their exceedingly small size and lowpotential power consumption are ideal for certain applications such asmixed reality systems. Exemplary FSPs are described in U.S. Utilitypatent application Ser. Nos. 14/738,877 and 14/555,585, the contents ofwhich have been previously incorporated-by-reference herein.

The embodiments disclosed herein describe projecting textured,patterned, or structured light with an FSP for use in depthreconstruction, as opposed to a typical “panel” type display (e.g., LCOSor DLP). The vast majority of existing active depth sensing systems usephoton sources such as LCOS and DLP, which project light with pixelatedpatterns (e.g., squares or rectangles). These systems illuminate theentire frame simultaneously, and the image projected is composed ofmultiple rectilinear (or fixed shape) display pixels projected at once.As a result, these images are composed of a plurality of repeated(generally rectilinear/square) shapes instead of clean lines.

FIG. 2 depicts a “line” 200 projected by a panel type display at amagnification sufficient to show imperfections in the projected line200. At this magnification, the projected line 200 appears as a seriesof right angles forming a set of steps. However, at lowermagnifications, the projected line 200 would appear linear to a viewer.Each of the arms of the right angles is formed by a linear series ofdisplay pixels.

As a result of the irregularities of the projected line 200, a sensor ofcomparable (or higher) resolution detecting the projected line 200 willhave sufficient sensitivity to observe high frequency noise in theprojected patterns, complicating the image reconstruction efforts. FIG.3 shows the projected line 200 from FIG. 2 with overlaid square-shapedscan areas 202 from a sensor with lower resolution than the projector.As shown in FIG. 3, the scan areas 202 along the projected line 200 donot contain identical amounts of the projected line 200. Therefore, thesignal from the sensor is very noisy, resulting in aliasing, i.e., awide grey line instead of a narrow black line.

This noisy signal problem complicates the task of locating theintersection 204 of two projected lines 200 a, 200 b, as shown in FIG.4. This intersection 204 may be a POI for a depth sensing system.However, the high frequency noise in the projected lines 200 a, 200 bresults in the intersection 204 being a line segment instead of a point.Further, the scan areas 202 are not capable of resolving the lines 200a, 200 b in sufficient detail to identify the intersection 204 at a highaccuracy.

Existing systems address this problem either by using a projector withmuch higher resolution than the corresponding sensor, or by usingadditional signal processing algorithms to reconstruct this noisiersignal back into something suitable for sub-pixel (or even at-pixel)precision mapping.

The embodiments disclosed herein describe projecting light using a FSPto produce a higher quality dynamic pattern to facilitate imagereconstruction. When using FSPs, light is projected by a singletraveling beam (e.g., a laser). The beam is mechanically scanned acrossthe scene at relatively high frequencies. Rather than projecting lightonto substantially every pixel of the scene (as with panel typedisplays), FSPs create optically distinct patterns by projectingbeam-patterns with narrow dispersion angles onto the surface, such thatthe light projected minimizes aliasing (e.g., image warping or jaggies)and high-frequency noise, both of which interfere with reconstructionefforts.

Whereas panel type displays have fixed illumination patterns, FSPs aredynamic. While the effective scan frequency of the FSP might allow only200 separate X and Y travel paths (for example), the phase-offset,illumination pattern, and scan frequency can be varied, allowing adynamic pattern that provides clear non-aliasing edges, intersections,and thus easily identifiable POIs, without requiring an extremely highresolution projector.

For instance, FIG. 5 depicts a pattern 300 projected by an FSP. Thepattern is made by modulating the FSP (e.g., with an actuator) to “draw”two instances/passes of a sin wave 302 a, 302 b that are phase-offset.As such, the first and second sin waves 302 a, 302 b intersectperiodically, forming a regular set of intersections 304. The FSP mayproject light from a tight beam light source (e.g., a laser) to form thepattern 300 depicted in FIG. 5. Accordingly, the signal from the lighthas minimal high frequency noise and negligible aliasing. Theintersections 304 form particularly desirable POIs because intersectionsare more accurately and precisely identifiable by image analysis thanpoints, which will increase in diameter as light spreads, requiringestimation of the center of the point. Another FSP pattern that formdiscrete intersections is the Lissajous pattern.

FIG. 6 depicts another pattern 300′ projected by an FSP. The pattern300′ depicted in FIG. 6 is similar to the pattern 300 depicted in FIG. 5and described above. In fact, the pattern 300′ depicted in FIG. 6 isformed by modulating the FSP to form the exact same first and second sinwaves 302 a, 302 b shown in FIG. 5. However, the FSP is furthermodulated by activating the light source therein only when the first andsecond sin waves 302 a, 302 b cross to form intersections 304.Projecting this pattern 300′ with an FSP results in (1) more distinctiveand identifiable intersections 304 (POIs) and (2) reduced system energyusage from deactivating the light source.

FIG. 7 depicts still another pattern 300″ projected by an FSP. Thepattern 300″ depicted in FIG. 7 is almost identical to the pattern 300′depicted in FIG. 6 and described above. The first and second sin waves302 a′, 302 b′ shown in FIG. 7 are phase shifted (by the same amount)compared to the first and second sin waves 302 a, 302 b shown in FIG. 6.Accordingly, the intersections 304′ formed by the first and second sinwaves 302 a′, 302 b′ shown in FIG. 7 are also phase shifted compared tointersections 304 formed by the first and second sin waves 302 a, 302 bshown in FIG. 6.

The displacement of the intersections 304′ compared to the intersection304 is depicted in FIG. 8, which is FIG. 7 with the first and second sinwaves 302 a, 302 b and the intersections 304 (from FIG. 6) shown inshadow. Time sequentially switching between the pattern 300′ depicted inFIG. 6 (and in shadow in FIG. 8) and the pattern 300″ depicted in FIG. 7(and in solid in FIG. 8) causes the intersections 304/304′ to appear tomove. This movement results in a dynamic pattern with more distinctiveand identifiable intersections 304/304′ (POIs).

FIG. 9 depicts another pattern 300′″ projected by an FSP. The pattern300′″ depicted in FIG. 9 is similar to the pattern 300 depicted in FIG.6 and described above. The first and second sin waves 302 a″, 302 b″shown in FIG. 9 are modified compared to the first and second sin waves302 a, 302 b shown in FIG. 6. Accordingly, the intersections 304″ formedby the first and second sin waves 302 a″, 302 b″ shown in FIG. 9 have amodified shape compared the intersections 304 formed by the first andsecond sin waves 302 a, 302 b shown in FIG. 6. The shape of theintersections 304″ in FIG. 9 are +'s, whereas the shape of theintersections 304 in FIG. 6 are X's. However, the locations of theintersections 304, 304″ in FIGS. 6 and 9 are the same. Accordingly, timesequentially switching between the pattern 300′ depicted in FIG. 6 andthe pattern 300′″ depicted in FIG. 9 causes the intersections 304/304″to appear to change shape (between X's and +'s). This shape changeresults in a dynamic pattern with more distinctive and identifiableintersections 304/304″ (POIs).

The patterns 300, 300′, 300″, 300′″ depicted in FIGS. 5-9 depict thesame change to each intersection 304, 304′, 304″ (POI) in the patterns300, 300′, 300″, 300′″. In other embodiments, a subset of theintersections 304, 304′, 304″ may change (e.g., position, shape,wavelength, etc.). In still other embodiments various subsets of theintersections 304, 304′, 304″ may have different changes. For instance,only the intersection(s) 304, 304′, 304″ being used for depth sensingmay change. In other embodiments, the number of intersections 304, 304′,304″ may change dynamically from dense to sparse. In still otherembodiments, the light source (e.g., laser) can be pulsed to dynamicallyvary the patterns 300, 300′, 300″, 300′″.

FIG. 10 depicts an active depth sensing system 400 capable of projectingthe patterns 300, 300′, 300″ described above according to oneembodiment. The system 400 includes a spatially modulated lightprojection device 402 (e.g., an FSP), two light sensors 404 a, 404 b(e.g., cameras), and a processor 406 operatively coupled to the othercomponents 402, 404 a, 404 b. The spatially modulated light projectiondevice 402 (e.g., an FSP), the light sensors 404 a, 404 b (e.g.,cameras), and the processor 406 may be coupled by a bus (not shown) inthe system 400. Alternatively, some or all of these components 402, 404a, 404 b, 406 may be coupled to each other by a network (e.g., awireless network).

FIG. 11 depicts the active depth sensing system 400 depicted in FIG. 10in use. The spatially modulated light projection device 402 is modulatedto project a pattern 408 (e.g., +) onto a substantially homogenoussurface 410 (e.g., a blank wall). The pattern 408 can be used as a POIto determine the distance Y between the light projection device 402 andthe surface 410. The light projection device 402 projects light 412 ontothe surface 410 to form the pattern 408. Reflect light 412′, 412″ isdetected from the first and second light sensors 404 a, 404 b.

The system 400 measures angles α, ß by which the pattern 408 isdisplaced from the respective optical axes 414 a, 414 b of the first andsecond light sensors 404 a, 404 b. Using one of the measured angles α, ßand the known distances X₁, X₂ separating the light projection device402 and respective first and second light sensors 404 a, 404 b, thesystem 400 can calculate the distance Y between the light projectiondevice 402 and the surface 410. Having measured angles α, ß, the system400 can provide a more accurately and precisely calculated distance Y.

FIG. 12 depicts an active depth sensing system 400 according to anotherembodiment. The system 400 depicted in FIG. 12 is similar to the onedepicted in FIGS. 10 and 11. In addition to the spatially modulatedlight projection device 402, the two light sensors 404 a, 404 b and theprocessor 406, the system 400 depicted in FIG. 12 also includes anactuator 416 to modulate the light projection device 402. The processor406 of the system 400 depicted in FIG. 12 includes a pattern designer418 and a pattern detector 420 running thereon. The pattern designer 418generates patterns and sends data specifying the generated patterns tothe actuator 416, which modulates the light projection device 402 todisplay the generated patterns. The pattern detector 420 receivesoptical data from the first and second light sensors 404 a, 404 b andextracts information regarding the pattern from the received opticaldata.

Further, because the FSP simply acts as a spatially modulated conduitfor light beams, light beams having different wavelengths can be passeddown the FSP simultaneously. This allows not only invisible infraredlight to be transmitted, but also visible light of various colors. Usingmultiple light beams allows the FSP to augment an infrared pattern witha visible pattern, allowing correspondence between camera sensors thatmight not normally be able to see in similar wavelengths. This can aidin registration of the camera sensors to a common coordinate system,provide additional depth reconstruction information (allowing featuresfrom one sensor to help provide supplemental information to anothersensor). Such a system can also perform other function, such asassistive illumination to indicate a goal, a region for use in focusing,a warning, etc.

Active depth sensing systems including an FSP have the followingcharacteristics. FSPs project distinctive and dynamic (i.e., varyingover time) patterns, providing supplemental or improved information fordepth determination over time. Further, FSPs can tailor energy emissionto a focused area, thereby reducing power, and increasing energydelivered to a given area to overcome high frequency noise. FSPs alsohave minimal amounts of high frequency image noise, thereby simplifyingdepth determination calculations. Moreover, FSPs are able tosimultaneously project light from two light sources (e.g., infrared andinvisible light sources).

Further, while the active depth sensing systems described herein includeFSPs, other spatially modulated light projection devices can also beused in active depth sensing system while retaining the desirable systemcharacteristics. For instance, active depth sensing system can include amicro-electro-mechanical systems (“MEMS”) mirror scanner and a laserlight source. Like systems including FSPs, a system including a MEMSmirror scanner can project and scan a beam pattern over a surface. Inother embodiments, the system can also project and scan a pattern toform a portion of a variably-illuminated computer generated hologram.All spatially modulated light projection devices (e.g., FSPs and MEMSmirror scanners) project “beams” or “arcs” of light rather than displaypixels, and have the ability to vary the path and timing of those beams.

Image Sensor with Augmented Depth Sensing

Both passive and active depth sensing systems include at least onesensor (e.g., a camera) to detect light reflected from a surface. Asdescribed above, some depth sensing systems detect visible light (e.g.,ambient light), while others detect projected light patterns (e.g.,projected infrared light).

Existing depth sensing systems (passive and active) typically use twocamera sensors to detect visible and projected infrared light. Thesecamera sensors are each associated with different filters, such that onecamera sensor detects visible light (possibly with an RGB color filter),and the other camera sensor detects infrared light (by filtering outvisible light). This sensor arrangement is depicted in FIG. 13, whichdepicts a passive depth sensing system 500 including a visible lightsensor 502 and an infrared light sensor 504, both operatively coupled toa processor 506. The infrared light sensor 504 includes a light cutfilter 508, which prevents all light except infrared light from reachingthe infrared light sensor 504. Before data from the visible light andinfrared light sensors 502, 504 can be used together, the coordinatesystems of the sensors 502, 504 must be registered. Registration of thecoordinate systems is especially difficult when the depth sensing system500 is moving relative to the surface.

The embodiments disclosed herein describe a hybrid visible/full spectrumlight sensor for use in depth sensing systems. As used in thisapplication, “full spectrum light,” includes visible and infrared light(wavelengths from about 390 nm to about 1 mm). For instance, FIG. 14depicts a passive depth sensing system 600 according to one embodiment.The passive depth sensing system 600 includes a hybrid visible/fullspectrum light sensor 602 operatively coupled to a processor 604. Thevisible/full spectrum light sensor 602 includes a hybrid filter 606, asdescribed in detail below. The hybrid filter 606 allows only visiblelight to reach some portions (i.e., sensor pixels) of the visible/fullspectrum light sensor 602, but allows full spectrum light to reach otherportions (i.e., sensor pixels) of the visible/full spectrum light sensor602.

FIG. 15 a passive depth sensing system 600 according to anotherembodiment. The passive depth sensing system 600 includes a hybridvisible/full spectrum light sensor 602 operatively coupled to aprocessor 604. The visible/full spectrum light sensor 602 includes ahybrid filter 606, as described in detail below. The processor 604includes an image processor 608 running thereon.

FIG. 16 schematically depicts a portion of a hybrid visible/fullspectrum light sensor 602 and its hybrid filter 606 for use in thepassive depth sensing systems 600 like those depicted in FIGS. 14 and15. The portion of the hybrid visible/full spectrum light sensor 602depicted in FIG. 16 include five sensor pixels 610. The underlyingsensor pixels 610 are identical in so far as they can each detect fullspectrum light, i.e., visible light (wavelengths from about 390 nm toabout 700 nm) and infrared light (wavelengths from about 700 nm to about1 mm). However, the light reaching some sensor pixels 610 is modified bythe filter 606 disposed between a light source (e.g., a reflectingsurface) and the sensor 602.

As shown in FIG. 16, the filter 606 includes sub-filters or “caps” F1and V1-V4 disposed over corresponding sensor pixels 610. Full spectrumlight (“F-type”) caps allow full spectrum light to reach the sensorpixel 610 underlying the F-type caps. In fact, F-type caps may not haveany filtering ability at all. Visible light (“V-type”) caps allow onlyvisible light to reach the sensor pixel 610 underlying the V-type caps.Sensor pixel/cap 610F1 is surrounded by four sensor pixels/caps610V1-610V4 at cardinal positions, forming a “+” sign with sensorpixel/cap 610F1 in the middle. Accordingly, the hybrid visible/fullspectrum light sensor 602 includes a full spectrum light (“F-type”)sensor pixel surrounded by four visible light (“V-type”) sensor pixelsin cardinal positions relative to the full spectrum light sensor pixel.The hybrid visible/full spectrum light sensor 602 is different fromother sensors because of the combination of visible and full spectrumlight sensors pixels in one sensor and the arrangement of those sensorpixels.

While the sensor 602 and filter 606 depicted in FIG. 16 includes fivesensor pixels 610 arranged in a cross-shaped configuration, otherembodiments include configurations with different numbers of pixeland/or different shapes. The sensor data analysis methods describedherein can be modified to analyze data from sensors and filters withdifferent configurations.

FIGS. 17 and 18 schematically depict larger portions of the hybridvisible/full spectrum light sensor 602 depicted in FIG. 16 and itshybrid filter 606. As shown in FIGS. 17 and 18, V-type and F-type sensorpixels are disposed in an alternating pattern, such that for each sensorpixel (V-type or F-type), all four of its cardinal neighbors are of thecomplementary type.

In such a configuration, for a hybrid visible/full spectrum light sensor602 with P total sensor pixels (e.g. 640×480=307,200), half of thesensor pixels would be V-type, and half would be F-type. When each typeof sensor pixel is considered independently without further imageprocessing, this sensor pixel arrangement results in reduced effectiveresolution for sensor pixel type and application. Increasing the overallresolution sensor to compensate for this problem would increase powerconsumption, sensor size, and other system costs.

Depth sensing systems 600 including hybrid visible/full spectrum lightsensors 602 according to various embodiments compensate for reducedeffective resolution by image processing (using the image processor 608depicted in FIG. 15). While these methods may not recover 100% of theresolution of a separate visible or infrared sensor of comparable size,these methods can recover effectively higher resolution that would beexpected from the actual sensor pixel resolution of the hybridvisible/full spectrum light sensor 602 (e.g., 50% visible sensor pixelsand 50% full spectrum sensor pixels). Embodiments of image processingmethods for use with hybrid visible/full spectrum light sensors canrecover from about 50% to about 100% of the resolution of separatesensors. Other embodiments can recover about 66% to about 90% of theresolution. Still other embodiments can recover about 75% of theresolution.

The light used in the depth sensing system 600 includes wavelengths thatoverlap in the visible spectrum because all sensor pixels 610 detectvisible light. F-type sensor pixels 610F also detect infrared light. Inthe embodiment shown in FIGS. 16-18, each sensor pixel 610 has eightadjoining neighbors, four of each type. For example, as shown in FIG.17, 610F1 has four adjoining V-type sensor pixels 610V1-610V4 in thecardinal directions. 610F1 also has four adjoining F-type sensor pixels610F2-610F5 in the inter-cardinal directions.

The following is a general description of an image processing methodaccording to one embodiment. The F value of all F-type sensor pixels610F can be initially approximated as being comprised of two values—acalculated visible value (“CV”), and a calculated infrared value (“CI”).In a first pass, the measured visible light values (“V”) for the V-typesensor pixels 610V adjacent a F-type sensor pixel 610F are used toestimate a first pass CV for the F-type sensor pixel 610F. Then thefirst pass CV is used to estimate a first pass CI for the F-type sensorpixel 610F. In a second pass, the first pass CI is used to (moreaccurately) estimate a second pass CV for the F-type sensor pixel 610F.

For surfaces with substantially homogenous or well-understood CIs, thesecond pass CV can be estimated by subtracting the first pass CI from Ffor the F-type sensor pixel 610F, thereby providing information notavailable from the V-type sensor pixels 610V alone. Surfaces withsubstantially homogenous or well-understood CIs can be found in anindoor room with no additional infrared light source, and with anyinfrared light projector in the system disabled. Similarly, in anoutdoor environment, ambient sunlight typically provides largely diffuseillumination on each surface (based on that surface's infraredreflectance), and thus, the infrared illumination for each surface islargely homogenous or predictable.

In the case of a modulated infrared projector (as described above),information regarding a projected infrared pattern can be used tocalculate an effective resolution of a visible light image that ishigher than the number of V-type sensor pixels 610V (e.g., P/2) in thesensor 602. Information regarding a dynamically altering projectedinfrared pattern in certain regions of the surface (as described above)can also be used to calculate an increased effective visible light imageresolution. Further, the sensor 602 may be use as a full resolutionvisible light sensor when all sources of infrared light are eliminatedfrom the environment (e.g., indoor with no infrared sources).

Other methods according to various embodiments can include more than twopasses to increase the accuracy of the estimated CV and CI. Still otherembodiments use other measured that calculated values to estimated CVand CI for an F-type sensor pixel 610F. For instance, using dataacquired by the hybrid visible/full spectrum light sensor 602 depictedin FIG. 17, CV and CI of 610F1 can be estimated using CV and CI of610F2-610F5 in addition to V of 610V1-610V4. FIG. 18 shows that the610F1 is surrounded by even more F-type and V-type sensor pixels thatcan provide even more optical data for estimation of CV and CI of 610F1.One factor in estimation methods using all F-type and V-type sensorpixels is that the sum of F for all F-type sensor pixels 610F should beat least as bright (if not brighter) than the sum of V for all V-typesensor pixels 610V.

Similar processes can be used to calculate the CI of F-type sensorpixels 610F in the sensor 602, and generate an estimated infrared lightonly image, despite having no dedicated infrared-only pixels. Forinstance, FIG. 19 depicts an image processing method 700 for generatingseparate visible and infrared images from optical data acquired by asingle hybrid visible/full spectrum light sensor 602 according to oneembodiment.

At step 702, the depth sensing system 600 receives light reflected froma surface. The hybrid visible/full spectrum light sensor 602simultaneously receives visible light at each V-type sensor pixel 610Vand full spectrum light at each F-type sensor pixel 610F.

At step 704, the depth sensing system 600 determines a visible lightvalue (“V”) for each V-type sensor pixel 610V based on the visible lightreceived by the pixel 610V. At step 704, the depth sensing system 600also determines a full spectrum light value (“F”) for each F-type sensorpixel 610F based on the full spectrum light received by the pixel 610V.

At step 706, the depth sensing system 600 (e.g., the image processor608) calculates a calculated visible light value (“CV”) for each F-typesensor pixel 610F. The image processor 608 can calculate CV using the Vfor the V-type sensor pixels 610V adjacent to each F-type sensor pixel610F. In a simple embodiment, the Vs of the four V-type sensor pixels610V adjacent to an F-type sensor pixel 610F are averaged to generateCV. For example, in the hybrid visible/full spectrum light sensor 602depicted in FIG. 16, the Vs of 610V1-610V4 are averaged to generate a CVfor 610F1. This embodiment is most accurate for homogenous surfaces(e.g., a white wall), but accuracy drops off as surfaces become moreheterogonous.

In another embodiment, edge detection and gradient detection can beperformed on the opposite facing neighbors to determine a more accurateCV by taking into account non-homogenous surfaces including edges. Forexample, in the hybrid visible/full spectrum light sensor 602 depictedin FIG. 16, edge detection and gradient detection can be performed onthe Vs of 610V1/610V3 and 610V2/610V4. If a large gradient is foundbetween the Vs of 610V1/610V3, those Vs may be given less weight whengenerating a CV for 610F1. This embodiment is more accurate forheterogonous surfaces (e.g., a corner including an edge).

At step 708, the depth sensing system 600 (e.g., the image processor608) calculates a calculated infrared light value (“CI”) for each F-typesensor pixel 610F. The image processor 608 can calculate CI using the CVfor the F-type sensor pixel 610F. In a simple embodiment, CI iscalculated by subtracting CV from F for the F-type sensor pixel 610F.

At step 710, the depth sensing system 600 (e.g., the image processor608) generates a visible light image using the Vs from the V-type sensorpixels 610V and the CVs calculated for the F-type sensor pixels 610F. Atstep 712, the depth sensing system 600 (e.g., the image processor 608)generates an infrared light image using the CIs calculated for theF-type sensor pixels 610F. The visible light image, the infrared lightimage, or both can be used for depth sensing.

Further, demosaicing and/or edge sharpening algorithms can optionally beapplied to the image data to resolve the visible light value of theV-type sensor pixels 610V and the full spectrum light value of theF-type sensor pixels 610F prior to step 704. Resolving these lightvalues V, F increases the accuracy of the method 700.

The systems 600 described herein use optical data from a single hybridvisible/full spectrum light sensor 602 to generate a visible light imagewith an effective resolution higher than P/2, and similarly, an infraredlight image at approximately a lower resolution, on the same physicalsensor 602. As such, the light used to generate the visible and infraredlight images will pass through the same lens-stack, so any minorimperfections are reflected in both the visible and infrared lightimages. This arrangement has two further advantages over existingsystems. First, the visible and infrared light images will be taken fromexactly the same optical vantage point, giving a perfect 6-DOFcorrespondence for the focal point of the respective images. Thisobviates the need for registration and calibration, which is anespecially difficult problem for systems including separate camerasoperating on different wavelength spectra, as the cameras may not beable to detect the same calibration targets. Further, registration andcalibration introduces an additional source of error that decreases thesub-pixel accuracy of any later-stage operations performed on the imagedata. Second, the exact same sensor is used, and as such, exposure timeson the two images are perfectly synchronized. For images of surfaces inrelative motion (of the sensor, the surface or both), the visible andinfrared light images match temporally (even in the microsecondtimescale) as well as geometrically, allowing for more precise anddetailed analysis of the image.

In another embodiment, depicted in FIG. 20, the hybrid visible/fullspectrum light sensor 802 includes two types of sensor pixels 810:A-type and B-type. A-type and B-type sensor pixels 810A, 810B bothdetect both visible and infrared light, but in different proportions. Inone embodiment, A-type sensor pixels 810A may detect 75% of the visiblelight (“pV_A”) and 25% of the infrared light (“pI_A”) impinging on thepixel 810A. In that embodiment, B-type sensor pixels 810B may detect 60%of the visible light (“pV_B”) and 40% of the infrared light (“pI_B”)impinging on the pixel 810A. While the pV and pI components in thisembodiment add up to 100%, in other embodiments, the pV and pIcomponents can add up to more or less than 100%. For instance, theF-type (full spectrum) sensor pixels 610F in the hybrid visible/fullspectrum light sensor 602 depicted in FIGS. 16-18, and described abovehave pV=100% and pI=100%.

In such sensors 802, each sensor pixel 810 has a sensed value (“SV”)corresponding to the detected visible and infrared light. Because SV fora particular A-type sensor pixel 810A (“SV_A”) is composed of twocontributors (i.e., total visible light value “V” and total infraredlight value “I” as modified by the proportion of each type of lightdetected by the sensor pixel 810A), we know that SV_A=(V*pV_A)+(I*pI_A).

Each A-type sensor pixel 810A also has an estimated value (“EV_A”)calculated using optical data from the adjacent sensor pixels 810 (e.g.,cardinal neighbors). For instance, 810A1 has an EV calculated from theSV for 810B1-810B4. In other words: EV_A=f ((V*pV_B)+(I*pI_B)) for810B1-810B4. The function f can be as simple as averaging. In otherembodiments, the function f may include edge detection and gradientdetection, as described above.

SV_A is determined by the sensor pixel 810A and EV_A is estimated. pV_A,pI_A, pV_B, pI_B are known from the design of the sensor 802. With thesedetermined, estimated and known values, the two equationsSV_A=(V*pV_A)+(I*pI_A) and EV_A=f ((V*pV_B)+(I*pI_B)) can be solved forV and I for each A-type sensor pixel 810A. A similar process can be usedto determined V and I for each B-type sensor pixel 810A.

FIG. 21 depicts an image processing method 900 for generating separatevisible and infrared images from optical data acquired by a singlehybrid visible/full spectrum light sensor 802 according to oneembodiment.

At step 902, the depth sensing system receives light reflected from asurface. The hybrid visible/full spectrum light sensor 802simultaneously receives full spectrum light at each sensor pixel 810(both A-type and B-type), with the percentage of visible and infraredlight detected dependent on the pV and pI of each A-type and B-typesensor pixel 810.

At step 904, the depth sensing system determines a sensed light value(“SV”) for each sensor pixel 810.

At step 906, the depth sensing system calculates a total visible lightvalue (“V”) and a total infrared light value (“I”) for each sensor pixel810. For instance, the depth sensing system can calculate V and I foreach sensor pixel by simultaneously solving the pair of equationsdescribed above (i.e., SV_A=(V*pV_A)+(I*pI_A) and EV_A=f((V*pV_B)+(I*pI_B))) using the known pV and pI values and the detectedSV and estimated EV values, as described above.

At step 908, the depth sensing system generates a visible light imageusing the calculated Vs for the sensor pixels 810. At step 910, thedepth sensing system generates an infrared light image using thecalculated Is for the sensor pixels 810. The visible light image, theinfrared light image, or both can be used for depth sensing.

While the method 900 depicted in FIG. 21 generates first a visible lightimage then an infrared light image, in other embodiments, a depthsensing system may generate first an infrared light image, then avisible light image. In still other embodiments, the depth sensingsystem may generate either a visible light image or an infrared lightimage without generating the other image.

While the dynamic non-aliasing pattern projector and image sensor withaugmented depth sensing are described as part of one system in someembodiments, the projector and sensor are independent and each canfunction with all of the described benefits without the other.

While the above-referenced sensors are described as depth sensing,sensors according to the embodiments can be used in othervisible/infrared light systems, such as camera focusing systems. Theabove-described depth sensing systems are provided as examples ofvarious optical systems that can benefit from hybrid sensors.Accordingly, use of the optical systems described herein is not limitedto the disclosed depth sensing systems, but rather applicable to anyoptical system.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the invention. The specification and drawingsare, accordingly, to be regarded in an illustrative rather thanrestrictive sense.

The invention claimed is:
 1. A depth sensing system, comprising: aspatially modulated light projection device to project light toward asurface; an actuator to control spatial modulation of the lightprojection device to form first and second sin waves that arephase-offset from each other to form a plurality of intersecting lightsegments; and a processor to analyze the light reflected from thesurface to determine a depth of the surface, wherein the actuatorcontrols spatial modulation of the light projection device tosequentially change a subset of the plurality of the intersecting lightsegments to form a dynamic pattern.
 2. The system of claim 1, whereinthe light comprises infrared light having a wavelength from about 700 nmto about 1 mm.
 3. The system of claim 2, wherein the light comprisesvisible light having a wavelength from about 390 nm to about 700 nm. 4.The system of claim 3, wherein the actuator controls spatial modulationof the light projection device to form a pattern comprising the infraredlight and the visible light.
 5. The system of claim 1, wherein theactuator controls spatial modulation of the light projection device toform two or more intersecting line segments on the surface.
 6. Thesystem of claim 5, wherein the actuator controls spatial modulation ofthe light projection device to form a dynamic pattern comprising the twoor more intersecting line segments on the surface.
 7. The system ofclaim 5, wherein the actuator controls spatial modulation of the lightprojection device to form a pattern comprising a plurality of discreteintersecting line segments on the surface.
 8. The system of claim 1,wherein controlling spatial modulation of the light projection deviceincludes controlling movement of at least a portion of the lightprojection device.
 9. The system of claim 8, wherein controlling spatialmodulation of the light projection device includes controllingprojection of the light by the light projection device.
 10. The systemof claim 1, the spatially modulated light projection device comprising afiber scanned display.
 11. The system of claim 1, wherein controllingspatial modulation of the light projection device comprises varying aphase-offset or a scan frequency of the light projection device.
 12. Thesystem of claim 1, wherein the actuator controls spatial modulation ofthe light projection device to sequentially change the shape of thefirst and second sin waves to form the dynamic pattern.
 13. The systemof claim 1, wherein the actuator controls spatial modulation of thelight projection device to sequentially phase shift the light to form adynamic pattern.
 14. The system of claim 1, further comprising a lightsource to generate the light.
 15. The system of claim 14, wherein thelight source is a laser.
 16. The system of claim 14, wherein theactuator also controls activation the light source.
 17. The system ofclaim 16, wherein the actuator controls spatial modulation of the lightprojection device to form first and second sin waves, and wherein theactuator activates the light source only when the first and second sinwaves intersect with each other.
 18. The system of claim 1, wherein theprocessor comprises a pattern designer to generate patterns and to senddata specifying the generated patterns to the actuator.
 19. The systemof claim 1, wherein the processor comprises a pattern detector toextract information from the light reflected from the surface.